Computer Assisted Topic Classification

1 Running Head: COMPUTER ASSISTED TOPIC CLASSIFICATION PRE-PUBLICATION VERSION. Cite as JITP 4:4, Forthcoming. There are a few known changes to the colors of the graphs and there may be other editorial changes as suggested by the editors. Abstract Social scientists interested in mixed methods research have traditionally turned to human annotators to classify the documents or events used in their analyses. The rapid growth of digitized government documents in recent years presents new opportunities for research but also new challenges. With more and more data coming online, relying on human annotators becomes prohibitively expensive for many tasks. For researchers interested in saving time and money while maintaining confidence in their results, we show how a particular supervised learning system can provide estimates of the class of each document (or event). This system maintains high classification accuracy and provides accurate estimates of document proportions, while achieving reliability levels associated with human efforts. We estimate that it lowers the costs of classifying large numbers of complex documents by 80% or more. 3 Technological advances are making vast amounts of data on government activity newly available, but often in formats that are of limited value to researchers as well as citizens. In this paper, we investigate one approach to transforming these data into useful information. " Topic classification " refers to the process of assigning individual documents (or parts of documents) to a limited set of categories. It is widely used to facilitate search as well as the study of patterns and trends. To pick an example of interest to political scientists, a user of the Library of Congress' THOMAS website (http://thomas.loc.gov) can use its Legislative Indexing Vocabulary (LIV) to search for congressional legislation on a given topic. Similarly, a user of a commercial Internet service turns to a topic classification system when searching, for example, Yahoo! Flikr for photos of cars or Yahoo! Personals for postings by men seeking women. Topic classification is valued for its ability to limit search results to documents that closely match the user's interests, when compared to less selective keyword-based approaches. However, a central drawback of these systems is their high costs. Humans— who must be trained and supervised—traditionally do the labeling. Although human annotators become somewhat more efficient with time and experience, the marginal cost of coding each document does not really decline as the scope of the project expands. This has led many researchers …